4.3. Metrics
Why are ordinary software metrics insufficient?
Availability and latency still matter, but an agent can return HTTP 200 while selecting the wrong tool, inventing state, leaking PII, or taking an unapproved action. Agent quality therefore needs engineering, behavioral, safety, and operational signals.
Which metrics are real release gates here?
| Signal | Source | Gate |
|---|---|---|
| Static checks | mise run check |
Zero warnings/errors |
| Branch coverage | pytest-cov | At least 95% |
| Tool trajectory | ADK eval | Average score 1.0 for recorded cases |
| Adversarial regression | mise run redteam |
All deterministic cases pass |
| Complete conversation | MLflow code scorer | One non-empty response per expected turn |
| Response facts | MLflow code scorer | Stable domain/policy facts present in every expected turn |
| Write-tool policy | MLflow code scorer | Exact state-changing name, arguments, order, and count |
| Gateway judge | Optional MLflow scorer | Evidence only unless a release policy defines a threshold |
The live eval gates are manual/provider-dependent. They do not make offline CI nondeterministic.
Which operational indicators should you observe?
- End-to-end and model/tool span latency distributions.
- Request/error counts by agent, model, route, and tool.
- Input/output token usage and model call count.
- MCP/A2A/gateway rejection and rate-limit counts.
- Approval requests, denials, executions, and audit failures.
- Retrieval no-match rates and tool-error rates.
Chapter 7 shows which of these are emitted or derived by the shipped OTel pipeline. A desired metric is not claimed as implemented until a collector query or scorer produces it.
How should metrics be segmented?
Aggregate averages hide failures. Segment by prompt/eval case, model, prompt version, route, tool, status code, environment, and deployment revision. Never use raw user text or high-cardinality session ids as Prometheus labels.
How do you define a useful SLO?
Tie it to an observable user outcome. For example: "At least 99% of read-only A2A requests complete without a server or gateway error over 30 minutes" is measurable. "The agent is helpful" needs an evaluation rubric and feedback sampling plan before it can be an SLO.
What is the metrics checkpoint?
Create a one-page scorecard for one change: offline gates, exact eval set/model/prompt version, trajectory score, observed failures, latency distribution, and token/call count. If a cell has no data, mark it unknown rather than estimating it from prose.